Configure dbt Cloud Connection Details

dbt Cloud is a managed platform that provides a user-friendly, cloud-based interface for data teams to develop, test, and deploy data transformation SQL code. It offers features like a web-based IDE, built-in Git integration for version control, and job scheduling.

The Calibo Accelerate Platform supports dbt Cloud for custom transformation pipelines. You can import your existing projects into the platform and create and run a new job or run an existing job.

Configuring dbt Cloud connection details

1. Sign in to the Calibo Accelerate platform and click Configuration in the left navigation pane.
2. On the Platform Setup screen, on the Cloud Platform, Tools & Technologies tile, click Configure.
3. On the Cloud Platform, Tools & Technologies screen, in the Data Integration and Data Transformation section, click Configure.
(After you save your first connection details in this section, you see the Modify button here.)


4. On the dbt Cloud screen, do the following:

   1. In the Details section, provide the following details:

      Field	Description
      Name	Give a unique name to your dbt Cloud configuration. This name is used to save and identify your specific dbt Cloud connection details within the Calibo Accelerate platform.
      Description	Provide a brief description that helps you identify the purpose or context of this dbt Cloud configuration.

   2. In the configuration section provide the following details:

      Field	Description
      Account ID	Provide the dbt account ID.
      Base URL	Provide the dbt base URL.

   3. Depending on how you want to retrieve the credentials to connect to dbt Cloud, select one of the following Secret Manager options:

      Secret Manager	Additional Information
      Calibo	Provide the dbt Personal Token.
      AWS Secrets Manager	In the Secret Management dropdown list, the AWS Secrets Manager configurations (that you save and activate in the Secret Management section on the Cloud Platform, Tools & Technologies screen) are listed for selection. Do the following: Select the configuration of your choice. Provide the Secret Name used to store the dbt credentials. Provide the dbt Personal Token.
      Azure Key Vault	In the Secret Management dropdown list, the Azure Key Vault configurations that you save and activate in the Secret Management section on the Cloud Platform, Tools & Technologies screen are listed for selection. Do the following: Select the Vault Configuration of your choice. Provide the Vault Name used to store the dbt credentials. Provide the dbt Personal Token.

   4. Test Connection - Click Test Connection to verify the provided details and ensure that the Calibo Accelerate platform is able to successfully connect to the configured dbt Cloud instance.

5. In the Configuration section, select one of the following:
   Existing dbt Projects

   Select the existing dbt projects that you want to import and click Add.

   + New Project

   Provide the following information to create a new project:

   Project Name	Project Name - Provide a name for the new project that you want to create in dbt. Description - Provide additional information about the new project . Advanced Settings - Turn on this toggle to provide additional settings. Project Subdirectory - Provide the name of the file/path that contains your dbt project.
   Development Environment	Configure the development environment used by the dbt project. You can either use an existing connection or create a new connection: Existing Connection Select this option to use a previously configured dbt connection. You can choose the required connection from the following list: Snowflake Databricks Unity Catalog PostgreSQL Once you select the option, provide the following information: Connection from dbt - Select a connection from the dropdown. Schema Name - Provide the schema name for the selected connection. View Connection Details - click the link to view details like account, database, warehouse, client session keep alive, and role. Auth Method - The supported authentication method is username and password. Username - Provide the username for the dbt connection. Programmatic Access Token (PAT) - Provide the programmatic access token or PAT for the dbt connection. Test Connection - Click to test the existing dbt connection. New Connection Select this option to create a new development environment connection. Calibo Accelerate supports the following connection types: Snowflake Databricks Unity Catalog PostgreSQL You can either select an existing configuration or click + New Configuration to create a configuration. Note: When creating a new connection using an existing configuration, the platform automatically reuses the credentials stored in that configuration to establish the dbt connection. No additional credential configuration is required. To create a new connection, provide the following details: Connection Type - Select the type of connection from the dropdown: Snowflake Databricks Unity Catalog PostgreSQL Existing Configuration - Select an existing configuration from the dropdown. + New Configuration - Click this option to create a new configuration and provide the following details: Connection Name - Provide a name for the dbt connection. Schema Name - Provide a schema associated with the dbt connection. dbt builds your models in this schema. This schema must be unique to your personal development environment in Snowflake. Role - Specify a Snowflake role that dbt will assume for this connection. Session Keep Alive (Optional) - If checked, this option keeps Snowflake sessions alive beyond the 4 hour timeout limit.
   Repository	Configure the source code repository associated with the dbt project. Provide the following information: Repository Provider - Select the one of the following repository providers from the dropdown. Git Clone GitLab If you select Gitlab, provide the following details: Group - Select a group to which you want to add this repository. Groups help you to organize, manage, and share the repositories for the product. Repository - Select a repository for the new project. If you select Git Clone, provide the Git URL.
   Click Add.
   On successful creation of the project in Git Clone, an SSH key is generated. Copy the key. You must add this SSH key in your personal setting of the repository. In Bitbucket, navigate to Personal Bitbucket Settings > SSH keys and click Add Key. Provide a name for the key, paste the key in SSH Public Key and click Add Key. Note: If you do not copy the SSH key in your Personal Settings, it may result in job failure.

6. Secure configuration details with a password
   To password-protect your dbt connection details, turn on this toggle, enter a password, and then retype it to confirm. This is optional but recommended. When you share the connection details with multiple users, password protection helps you ensure authorized access to the connection details.

7. Click Save Configuration. You can see the configuration listed on the Data Integration and Data Transformation screen.

Recommended Topics

What's next? Data Transformation using dbt Cloud